An Evaluation of the Content Quality, Readability, and Reliability of Publicly Available Web-Based Information on Pneumothorax Surgery in Ireland

Introduction The internet is increasingly the first port of call for patients introduced to new treatments. Unfortunately, many websites are of poor quality, thereby limiting patients’ ability to make informed health decisions. Within thoracic surgery, the treatment options for pneumothoraces may be less intuitive for patients to understand compared to procedures such as lobectomies and wedge resections. Therefore, patients must receive high-quality information to make informed treatment decisions. No study to date has evaluated online information regarding pneumothorax surgery. Knowledge regarding the same may allow physicians to recommend appropriate websites to patients and supplement remaining knowledge gaps. Objective This study aims to evaluate the content, readability, and reliability of online information regarding pneumothorax surgery. Methods A total of 11 search terms including "pneumothorax surgery," "pleurectomy," and "pleurodesis" were each entered into Google, Bing, and Yahoo. The top 20 websites found through each search were screened, yielding 660 websites. Only free websites designed for patient consumption that provided information on pneumothorax surgery were included. This criterion excluded 581 websites, leaving 79 websites to be evaluated. To evaluate website reliability, the Journal of American Medical Association (JAMA) and DISCERN benchmark criteria were applied. To evaluate the readability, 10 standardized tools were utilized including the Flesch-Kincaid Reading Ease Score. To evaluate website content, a novel, self-designed 10-part questionnaire was utilized to assess whether information deemed essential by the authors was included. It evaluated whether websites comprehensively described the surgery process for patients, including pre- and post-operative care. Website authorship and year of publication were also noted. Results The mean JAMA score was 1.69 ± 1.29 out of 4, with only nine websites achieving all four reliability criteria. The median readability score was 13.42 (IQR: 11.48-16.23), which corresponded to a 13th-14th school grade standard. Only four websites were written at a sixth-grade reading level. In the novel content questionnaire, 31.6% of websites (n = 25) did not mention any side effects of pneumothorax surgery. Similarly, 39.2% (n = 31) did not mention alternative treatment options. There was no correlation between the date of website update and JAMA (r = 0.158, p = 0.123), DISCERN (r = 0.098, p = 0.341), or readability (r = 0.053, p = 0.606) scores. Conclusion Most websites were written above the sixth-grade reading level, as recommended by the US Department of Health and Human Services. Furthermore, the exclusion of essential information regarding pneumothorax surgery from websites highlights the current gaps in online information. These findings emphasize the need to create and disseminate comprehensive, reliable websites on pneumothorax surgery that enable patients to make informed health decisions.


Results
The mean JAMA score was 1.69 ± 1.29 out of 4, with only nine websites achieving all four reliability criteria.The median readability score was 13.42 (IQR: 11.48-16.23),which corresponded to a 13th-14th school grade standard.Only four websites were written at a sixth-grade reading level.In the novel content questionnaire, 31.6% of websites (n = 25) did not mention any side effects of pneumothorax surgery.Similarly, 39.2% (n = 31) did not mention alternative treatment options.There was no correlation between the date of website update and JAMA (r = 0.158, p = 0.123), DISCERN (r = 0.098, p = 0.341), or readability (r = 0.053, p = 0.606) scores.

Conclusion
Most websites were written above the sixth-grade reading level, as recommended by the US Department of Health and Human Services.Furthermore, the exclusion of essential information regarding pneumothorax surgery from websites highlights the current gaps in online information.These findings emphasize the need to create and disseminate comprehensive, reliable websites on pneumothorax surgery that enable patients to make informed health decisions.

Introduction
The internet is often the first port of call for patients exploring new diseases and treatment options [1,2].However, the quality of web-based health information is often unregulated and at risk of lacking appropriate readability, reliability, and content.
Previous studies [3,4] have indicated that online health materials are written beyond the sixth-grade reading level recommended by the United States Department of Health and Human Services [5].This may lead to poor health literacy among patients from less educated backgrounds.The readability of online health materials can be assessed using several validated readability tools [6].
Previous studies evaluating web-based information on stoma bags and pneumothoraces have indicated that the reliability of this information is poor [7][8][9], with the average website meeting only half of the Journal of the American Medical Association (JAMA) Benchmark reliability criteria [10].Website reliability may also be assessed using other validated tools such as the DISCERN instrument [11].
To evaluate website content, several studies have designed novel content-scoring criteria for medical conditions including diabetic retinopathy and stoma bags [8,12].However, these criteria are disease-specific and therefore cannot be extrapolated or compared with other studies.No broadly applicable, validated content criterion is available in the literature.Therefore, it is difficult to compare the content quality of online health literature.
This is a rare study that has attempted to evaluate web-based patient information on pneumothorax surgery.The objectives of this study are to quantitatively assess the readability, reliability, and content of the most encountered websites containing information on pneumothorax surgery.

Search strategy
A cross-sectional review of the most encountered websites concerning pneumothorax surgery was performed on June 10, 2023, at 10 am in Cork, Ireland, by two authors (M.H. and P.H).Both authors cleared their search history, cache, and cookies before the search.

Inclusion and exclusion criteria
Websites that were in English, free to access, contained information regarding pneumothorax surgery, and designed for public consumption (not for health professionals) were included.
Websites with paywalls, social media websites, paid adverts, video-only websites, and websites that did not discuss pneumothorax surgery were excluded.

Data collection
The following website characteristics were extracted from each website: the year in which the website was last updated and the authorship of the website (government-affiliated, academic, non-profit, media, or private).The entire website text was also extracted for readability, reliability, and content criterion analysis by two evaluators (M.H. and S.A.).

Readability
We utilized 10 online, freely available tools to assess website readability [6]: the Flesch-Kincaid Grade Level, the Flesch-Kincaid Reading Ease Score, the New Dale-Chall Score, the Spache Readability Score, the Simple Measure of Gobbledygook (SMOG) Index, the Coleman-Liau Index, the Gunning-Fog Score, the Automated Readability Index, the Linsear Write Score, and the Rix Score.
They analyzed elements of a text including word count and sentence length.They each yielded a numerical score that corresponded to a school grade reading level.Additionally, an overall median readability score for each website was included.

Reliability
We utilized two freely available, validated instruments (the JAMA Benchmark Criteria and the DISCERN instrument) to assess website reliability.Permission was not required to use these instruments.
The JAMA criteria (Table 1) evaluated the presence of four criteria to quantify website reliability.A score of +1 was given for each criterion fulfilled.This resulted in a score between 0 and 4 with a higher score corresponding to higher reliability.To evaluate website reliability, the author utilized the freely available Journal of the American Medical Association (JAMA) Benchmark Criteria [10].
Permission was not required to utilize this questionnaire.Please find the link to the original author's website at https://jamanetwork.com/journals/jama/article-abstract/415407 The DISCERN instrument (Table 2) consisted of 16 questions evaluating website reliability.Each question was scored on a scale from 1 to 5. A higher score of 5 indicated the quality criterion had been completely fulfilled, whereas a lower score of 1 indicated the criterion was not fulfilled at all.The maximum total score was 80.
Questions Score Based on the answers to all of these questions, rate the overall quality of the publication as a source of information about treatment choices 1 2 3 4 5

TABLE 2: The DISCERN Instrument
To evaluate website reliability, the author utilized the freely available DISCERN instrument [14].Permission was not required to utilize this questionnaire.A link to the original author's website is available at http://www.discern.org.uk/discern_instrument.php

Content
As there was no validated content criterion available in the literature, a novel, 10-part "Yes or No" questionnaire was designed by the authors (M.H. and P.H.) to evaluate website content (Table 3).It evaluated whether websites comprehensively described the surgery process for patients including pre-and post-operative care.The questionnaire can be modified to evaluate the content of any website examining surgical procedures.

Statistical analysis
Website evaluation was performed by two authors (M.H. and S.A.).All statistical analysis was performed using Statistical Package for the Social Sciences (SPSS) version 28 (IBM Corp., Armonk, NY).Data was described using descriptive statistics (number, percentage, median, mean, and standard deviation).
Pearson's correlation coefficient was used to determine the association between questionnaire scores and the date of update of the website.A one-way ANOVA test was used to quantify the association between website authorship category and readability/reliability scores.Interobserver reproducibility was measured using the interclass correlation coefficient.

Authorship
In total, 32 (40.5%) websites were private, 34 (43%) were non-profit, eight (10.1%) were governmentaffiliated, four (5.1%) were academic, and one (1.3%) was media.There was a statistically significant difference in the DISCERN scores between websites of different authorships (p = 0.007) (Figure 1).This was not seen in the JAMA (p = 0.197) or readability (p = 0.252) scores.The overall median readability score for all websites was 13.42 (IQR: 11.48-16.23).This corresponded to a 13-14th school grade or a first-year university student's reading level.Four (5.16%) websites were written at the sixth-grade reading level recommended by the United States Department of Health and Human Services.Of the 10 readability tools used, six of them rated the mean readability scores of websites as being at least "fairly difficult to read" (Table 4).

Reliability analysis
We utilized the JAMA Benchmark Criteria and the DISCERN instrument to evaluate website reliability.
The mean JAMA reliability score was 1.69 (±1.29) out of 4. In total, 17 websites did not fulfill any of the JAMA criteria, while nine websites fulfilled all four criteria (Table 5).Notably, 50 (63.39%)websites did not display attributions/references.The interobserver reproducibility was r = 0.592 (p < 0.001).
Notably, the weakest performing section within the questionnaire was, "Does it describe what would happen if no treatment is used?," with a mean score of 2.48 (±1.55).The interobserver reproducibility was r = 0.387 (p = 0.017).

Content analysis
The mean number of content criteria met was 7 (±2.18)out of 10.Notably, 31.6% of websites did not mention side effects/complications of surgery, while 39.2% of websites did not mention alternative treatment options (Table 6).The interobserver reproducibility was r = 0.601 (p < 0.001).

Discussion
The internet has become an accessible tool for patients who wish to access information regarding diseases and treatment options.This study represents the first and most comprehensive evaluation of web-based information concerning pneumothorax surgery to date.A total of 660 websites were identified and 79 were evaluated.
The overall median readability score of our websites corresponded to a 13-14th school grade reading level, which is beyond the reading ability of the general population [15].Only four websites were written at the sixth-grade reading level recommended by the United States Department of Health and Human Services [3,4].Furthermore, six of the 10 individual readability tools used rated the mean readability scores of websites to be at least "fairly difficult to read".Previous studies have shown websites concerning pancreatic cancer and diabetic retinopathy [12,16] to also be written above this recommended level.This indicates that patients do not understand a substantial portion of the information they are reading.This may reflect a wider trend of clinician authors not explaining concepts in a language familiar to patients.Websites should be written in a simple, understandable manner to not confuse patients further, thus facilitating informed decision-making.
Our reliability analysis showed that the mean JAMA score of our websites was 1.69 (±1.29) out of 4, with 63.3% of websites not displaying references.This concerning lack of website source transparency hinders the patients' ability to make informed health decisions and potentially exposes the reader to misinformation.Additionally, our JAMA scores were lower than those from previous research evaluating web-based stoma information [8].However, our mean DISCERN score for websites was 57.18 (±10.47)out of 80, which was slightly higher than the same study evaluating web-based stoma information [8].These inconsistent results highlight the current challenges in accurately quantifying the reliability of web-based health information and reinforce the need for standardization of these types of questionnaires.To improve website reliability, search engines may recommend reliability checklists for websites.Completing these checklists could enhance the site's visibility.
The mean number of content criteria met was 7 (±2.18)out of 10 with many websites omitting important information regarding pneumothorax surgery.These included not mentioning the side effects of surgery (31.6% of websites) and not mentioning alternative treatment options (39.2% of websites).This may have occurred because website authors assumed that patients were already informed about these matters by their doctor.Nevertheless, these results are concerning, as an apparent lack of alternative treatment options may impact a patient's ability to give informed consent.We designed this novel content criterion because we could not find an existing questionnaire that could successfully evaluate the content of websites concerning pneumothorax surgery.As a result, this questionnaire was not previously validated, and the results cannot be directly compared with previous studies that have created content questionnaires.However, it may be modified and contributed to a future validated content questionnaire that applies to all websites examining surgical procedures.

Limitations
Our study had several limitations.First, the study excluded video platforms such as YouTube and social media due to the complexity of analyzing several information modalities within a single study.These sites represent a significant source of patient health information [17].Furthermore, this study only evaluated results from the top three search engines, thereby excluding all other search engines and a potential subset of patients who use these search engines.However, the search engines used make up 95% of all internet searches.
Additionally, despite our efforts to be comprehensive with our search strategy by including both esoteric and patient-friendly search terms, our search likely missed several websites.Furthermore, while search engines generally prioritize newer websites within the search engine algorithm, the authors did not exclude older websites with potentially outdated information.
Moreover, the DISCERN instrument was subjectively scored on a "one to five" scale, thus risking evaluator bias in the results.To mitigate this, two authors analyzed websites, and only the mean aggregate scores were included in the results.Similarly, two authors conducted the search strategy and applied the inclusion/exclusion criteria.However, due to the subjectivity of the DISCERN questionnaire, the interclass correlation coefficient was low (r = 0.387).This may limit the reliability of this section of the results.This was seen to a lesser extent with the JAMA (r = 0.592) and the content analysis (r = 0.601) as these were "Yes or No" questions.Refinement and standardization of the original DISCERN instrument may enhance the consistency of future studies' results.

Conclusions
This is one of the rare studies available in the literature to evaluate web-based pneumothorax surgery information.Most websites were written beyond the comprehension of the general population, hindering effective patient education and decision-making.Furthermore, many websites failed to disclose key information regarding pneumothorax surgery and failed to include sources for their information.These findings underscore the need for comprehensive, reliable websites on pneumothorax surgery that enable patients to make informed health decisions.

FIGURE 1 :
FIGURE 1: The mean DISCERN scores of websites from each authorship category

TABLE 5 : Number of JAMA criteria met by websites
JAMA: Journal of the American Medical Association Benchmark Criteria.